Naive Parallelization of Coordinate Descent Methods and an Application on Multi-core L1-regularized Classification

It is well known that a direct parallelization of sequential optimization methods (e.g., coordinate descent and stochastic gradient methods) is often not effective. The reason is that at each iteration, the number of operations may be too small. In this paper, we point out that because of the skewed distribution of non-zero values in real-world data sets, this common understanding may not be true if the method sequentially accesses data in a feature-wise manner. Because some features are much denser than others, a direct parallelization of loops in a sequential method may result in excellent speedup. This approach possesses an advantage of retaining all convergence results because the algorithm is not changed at all. We apply this idea on a coordinate descent (CD) method for L1-regularized classification, and explain why direct parallelization should work in practice. Further, an investigation on the shrinking technique commonly used to remove some features in the training process shows that this technique helps the parallelization of CD methods. Experiments indicate that a naive parallelization achieves better speedup than existing methods that laboriously modify the algorithm to achieve parallelism. Though a bit ironic, we conclude that the naive parallelization of the CD method is the best and the most robust multi-core implementation for L1-regularized classification.